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Abstract 

We investigate the expressive power of first-order quantifications in the context of 
monadic second-order logic over pictures. We show that k + 1 set quantifier alterna- 
tions allow to define a picture language that cannot be defined using k set quantifier 
alternations preceded by arbitrarily many first-order quantifier alternations. 

The approach uses, for a given picture language L and an integer m > 1, the 
height-m fragment of L, which is defined as the word language obtained by consid- 
ering each picture p of height m in L as a word, where the letters of that word are 
the columns of p. 

A key idea is to measure the complexity of a regular word language by the group 
complexity of its syntactic monoid. Given a picture language L, such a word language 
measure may be applied to each of its height fragments, so that the complexity of the 
picture language is a function that maps each m to the complexity of the height-m 
fragment of L. The asymptotic growth rate of that function may be bounded based 
on the structure of a monadic second-order formula that defines L. 

The core argument for that lower bound proof is based on Straubing's algebraic 
characterization of the effect of first-order quantifiers on the syntactic monoid of word 
languages by means of Rhodes' and Tilson's block product. 

Keywords: Picture languages, monadic second-order logic, quantifier alternation, 
syntactic monoid, group complexity 
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1 Introduction 

In monadic second-order logic (MSO) over finite structures, formulas use first-order quan- 
tifications (ranging over elements of the universe) as well as set quantifications (ranging 
over sets thereof). |MT97l ISch971 IMat99l IMST021 IMat02] investigate the effect of the 



alternation of existential and universal set quantification and show that the depth of this 
alternation cannot be bounded without loss of expressive power. The proofs are done for 
a specific class of structures, namely pictures. 

The upper bound proofs in those papers show that very little use of set quantification is 
needed. The quantifiers that actually alternate are all first-order. Set quantification is 
needed for two purposes: The outermost set quantification establishes a specific, uniquely 
determined coloring, and the innermost set quantification is needed only to replace the 
horizontal ordering, which itself is not present in the logic. 

Thus all the formulas constructed in the upper bound proofs can be written in prenex 
normal form with a quantifier prefix of the form 

3*{ 3 ,v}*3*{ 3 ,v}*, 

where 3* denotes a block of existential set quantifiers, and v and 3 denote universal (or 
existential, respectively) first-order quantifiers. The question whether every MSO formula 
can be written in this form remains open, see Problem 15 .21 

This motivates the interest in the power of first-order quantification in the context of MSO. 
It has been studied in [AFS00[ IJM01] in the context of graphs. In [AFSOO] , the authors 
suggest the closed MSO alternation hierarchy, which is coarser and more robust than the 
ordinary MSO alternation hierarchy of [MT97J because it allows to intersperse first-order 
quantifiers "for free" between set quantifiers. The authors ask whether this hierarchy is 
strict — a question that is still open. 

In |JM01j . the authors develop a technique to infer new separation results dealing with the 
first-order closure. Specifically, they show the following: 

Theorem 1.1 ([JM01]) Let V,W C {3,V, a,v}*. Let S be a graph property definable 
with a quantifier prefix of a form in V but not with one of a form in W . Then there is 
another property definable with a quantifier prefix of a form in 3 w V but not with one of 
a form in {a, v}* W. 

The authors of [JM01] apply that theorem to show the following corollary (previously 
shown directly in [AFSOOJ). 

Corollary 1.2 There exists a graph property definable by a prenex normal form of type 
3*{a, v}*3*{a,v}* but not with one of type {3, v}*3*{a, v}*. 

In this paper, we focus on pictures (as opposed to arbitrary finite graphs). We show that 
the above corollary is true for pictures, too, thereby giving yet another proof for it (see 
Corollary 12. f 21 for the case k — 1). Besides we consider (as in [Mat99j) formulas that have 
a quantifier prefix of the form 

{ 3! v}*{3,V}*{ 3 ,v}*, 
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Figure 1: Hasse diagram 
Hasse diagram of the monadic second-order alternation hierarchy and the first-order 
closures hierarchy over pictures (k > 1). All inclusions are trivial. The dashed line 
indicates that the two classes are incomparable. 

The non-inclusions (1) (from [Sch97|) and (2) (from [Mat99]) are re-proved here (for non- 
trivial alphabets), see Corollary 12. Hi That corollary shows also the non- inclusion (3), 
which is new. Non-inclusion (4) has been shown for directed graphs in [AFSOO, JM01] 
and is here re-proved (for pictures over a non-trivial alphabet) as Corollary 12.121 



where the set quantifier block in the middle contains only k alternations, and compare 
their expressive power to formulas with a quantifier prefix of the form 

{3,V}*{ 3 ,v}*, 

where the set quantifier block contains only k + l alternations. The main result of this 
paper (Corollary 12. 1 1[) is that there is a formula of the latter kind that is not equivalent 
to any of the former kind. Once again, the formula constructed in the proof does not 
actually use k + 1 set quantifier alternations; it has only two set quantifier blocks — the 
k+l alternations stem from the first-order quantifier blocks in between these. 
The lower bound proof is based on the block product introduced in [RT89] . which, by 
[Str94] . allows to characterize the effect of first-order quantifiers on the syntactic monoid 
of word languages, see Lemma 14.31 The application to pictures (as opposed to words) 
follows the same approach as [MT971 IMST021 [Mat99| IMat02] : The common essential idea 
to show that a picture language L cannot be denned by a formula class T is the following: 
We consider the family (L[m]) m >i, where L[m] contains the pictures of height m. That 
so-called height-m fragment L[m] may be regarded as a word language over the m-fold 
Cartesian product of the alphabet. Then we show that, for a sufficiently large m, the 
complexity of that height-m fragment (wrt. some suitable complexity measure of word 
languages) is too high, so that L cannot be defined by a J-"-formula. 
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Typical complexity measures used in |MT97t IMST02t IMat99] are firstly the number of 
states needed for a recognizing non-deterministic finite automaton and, secondly, the length 
of the shortest word of a (unary) word language. This paper is the first one in which the 
used complexity measure is the group complexity of the syntactic monoid. 

2 Basic Notions 

2.1 Pictures 

Let T be a finite alphabet. A non-empty picture of size (m,n) over T (where m,n> 1) is 
an mxn-matrix over T. If p is a non-empty picture of size (m,n), we denote the length 
n (i.e., the number of columns) by \p\, the height m (i.e., the number of rows) by p, and 
the domain {1, ..,m}x{l, ..,n} by dom(p). The component at position (z, j) G dom(p) is 
denoted p{i,j). 

The set of non-empty pictures over T is denoted A set of non-empty pictures is called 

a picture language. 

If p and q are non-empty pictures with p = q, then the non-empty picture that results by 
appending q to the right of p is denoted pq. This partial operation is called the column 
concatenation. 

Picture languages must not contain the empty picture. Nevertheless, when we assemble 
picture languages by column concatenation, it is often convenient to have a neutral element. 
That is why we consider a special, distinct empty picture which we denote e and for which 
height, width, domain, and size are not defined. For every (empty or non-empty) picture 
p, we define pe = ep = p. 

Column concatenation is lifted to sets of pictures as usual. For every set L of pictures, the 
iterated column concatenation is defined by L° = {e} and L l+1 = L l L and every % > 0. 
The set of all non-empty pictures of height m over alphabet T is denoted r m ' + , and T" 1 '* 
abbreviates r m ' + U {e}. 

Let p be a non-empty picture over T and m = p. We frequently consider each column of p 
as a letter of the new alphabet T m . This way, we identify every non-empty picture p with 
a word of length \p\ over alphabet T m . For a set of pictures L and m > 1 we define the 
height-m fragment (denoted L[m}) as the set of these words over alphabet Y m . 

2.2 Pictures over Attributes Alphabets 

While in general the nature of an alphabet T is indifferent, it will be technically convenient 
to have certain notions for the case that V is of the form {0, 1} / for a finite set of so-called 
attributes. That means that each letter a G V is a mapping / — > {0, 1}. 
If a G {0, l} 7 and J C J, then restrj(a) = a f J is the restriction of a to a J-indexed family. 
The mapping restrj is an alphabet projection from {0, 1} 1 — > {0, 1} J , which is lifted to 
pictures and picture languages the usual way. 

The alphabet projection exj : {0, l} 7 — > {0, 1} 7 \ J is defined by exj = restr/y. 
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Furthermore, we define for every /j, G I the mapping 

pr^ : {0,1} J ^ {0,1}, a^a(fi). 

It is an alphabet projection, too, and it is lifted to pictures and picture languages the usual 
way. 

Typically, each attribute corresponds to a free variable in a formula (see next section). 
With regard to sentences (i.e., formulas without free variables), it is therefore consequent 
to allow also the empty attribute set, so by convention, {0, l} is some fixed singleton 
alphabet, and restr0 denotes the alphabet projection to that singleton alphabet. 



2.3 Monadic Second- Order Formulas 

We describe our conventions for formulas. We will be concerned with a fixed signature with 
two binary successor predicates Si, S 2 and with the specific class of structures associated 
to non-empty pictures. 

Let J, K be two disjoint sets of attributes, which we use as indices of variables. We use 
first-order variables x v with v G K and set variables with \x G J. Atomic formulas 
are of the form X^Xy (for /i G J and v G K), or SiX^x u , or S 2 x IJi x u , or x M = x v (for 
/j,, v G K). Formulas are assembled in the usual way using the boolean connectives as well 
as first-order quantification (3x u tp or Wx u p) and set quantification (3X M (^ or VX^ip). 



2.4 Pictures as Models 

Let J, K be two disjoint attribute sets. Set I = J U K. Let Unique I K be the set of those 
non-empty pictures p over alphabet {0, l} 1 such that for all v G K there is exactly one 
position G dom(p) with p(i,j){v) = 1. 

Let p G Uniquej K , say with size (m,n). Let ip be a formula with free set variables in 
{X M | fi G J} and free first-order variables in {x u | v G K}. To p, we associate the grid 
structure with universe {1, .., m}x{l, .., n} and an assignment (Xfy^j, [x^vaK to the free 
variables in the following way: 

• x l = {(hj) e dom(p) | p(i,j)(n) = 1}, 

• x^ is the unique G dom(p) with p(i,j){v) = 1. 
We write 

iff this assignment makes <p true in the structure with universe {1, ..,m}x{l, ..,n}, where 
the predicates Si and S*2 are interpreted as the vertical and horizontal successor relation, 
respectively, i.e., S2X1X2 asserts that £2 is the horizontal successor of x\. That means, 
p |= S2X1X2 iff there exist (i, j), + 1) G dom(p) such that x\ = and x\ = + 1). 
Another notation convention will be convenient. Let ip and p be as above. Suppose that 
fi, . . . , v n G K are attributes of first-order variables. If ai, . . . , a n G dom(p), then, by abuse 
of notation, we write 
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iff if is made true in the structure from above, where the assignment for the attributes 
{ui, . . . , v n } is provided by setting x v v . = cij for every j £ {1, .., n}. 

Let Modi^i'f) = {p Unique j K \ p \= if}. We write Mod(if) rather than Modi^if) 
if /, K are clear from the context; typically I (or K) is the set of those attributes that 
may appear as indices of free variables (or free first-order variables, respectively) of f, or 
any superset thereof. Indeed we have the following remark, which shows that adding an 
element to the attribute set does not make that much of a difference: 

Remark 2.1 Let I be an attribute set, K C I, and n £ I\K. If if is a formula with 
indexes of free variables in I\{fj,}, then 

Mod I>K (<p) = ex { ^ } (Mod AM ^(v9)). 

The concept of existential set quantification is captured by the alphabet projection on 
picture languages in the following sense, motivating the notation ex^} for the alphabet 
projection. 

Remark 2.2 Let I be an attribute set, K C I, and /j, £ I\K . If if is a formula with 
indexes of free variables in I, then 

Mod I \ {lx}jK (3X /Ji f) = ex {lx} (Mod I:K ((f)). 

Two formulas if,tp are equivalent iff Mod(if) = Mod{tp). Note that our notion of equiva- 
lence implicitly refers to the class of pictures and is thus coarser than logical equivalence. 
A formula ip defines a picture language L over alphabet {0, 1} J if f has no free first-order 
variables, the free set variables of <p are among (X M )^ e j, and Modj$(<f) = L. 
We use the following convention for variable substitution. Let if be a formula. Let 
Xi, . . . , X m (and x±, . . . , x n ) be set variables (or first-order variables, respectively). Note 
that we do not require that all free variables of if are among these, but typically, this is 
the case. 

We may write <f(Xi, . . . , X m , x±, . . . , x n ) instead of if in order to pick these variables 
for a later substitution. If we later write if(X[, . . . ,X' m ,x' x , . . . ,x' n ), for other variables 
X[, . . . , X' m , x[, . . . , x' n , then this denotes the formula that results from if by replacing each 
indicated variable from the first variable tuple by the respective variable from the latter. 
For example, if we introduce the formula if as (p(xi,X2), then by (f(x 2 ,Xi) we mean the 
formula that results from ip by exchanging the occurrences of x x and x 2 - 

Example 2.3 The first-order formula top(x) := —3y(Siyx) asserts for a position x that it 
is in the top row. Similarly, left(x) := -d^S^yx) and right(x) := -i3y(S2xy) assert that x 
is in the leftmost (or rightmost, respectively) column. □ 

Example 2.4 Let 

ip = \/x 1 Wx 2 ((S 2 x 1 x 2 A X cld xi) ->• X cld x 2 ). 
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Then X c i d is the only free set variable of The formula ip asserts that X c i d is closed under 
horizontal successors. In other words, for a non-empty picture p over alphabet {0, l}{ dd } 
we have that p \= tp iff every row of the picture pr c}d (p) is in 0*1*. Let 

<p = \/X cld (X cld x A if) -)■ X cld x). 

Then i/j asserts that position x' is right to x. More precisely: for a non-empty picture p 
and two positions (i',f) G dom(p) we have p, AMZiLil |= iff i = and j < j'. 

Consider the formula right(x) from the preceding example. Let 

if' = 3X c i d (^X c i d x /\\/x\(right(xx) A x\^x' — > ~^X c i d xi) A 
Wx\x 2 [S2X1X2 A x\^x' A X c jdXi —)• X c idX2) J. 

Then is equivalent to y?. □ 

The formula ip from Example 12.41 will be abbreviated as x <2 x' and will be needed later. 
The above is a standard example of how to use set quantification to express the horizontal 
ordering, which we do not have in the signature. 



2.5 Quantifier Alternation Classes 

In this section we define the formula classes that are characterized by the structure of 
their prenex normal form wrt. blocks of existential or universal set quantifiers or first-order 
quantifiers. 

A first-order formula is a formula that does not make use of set quantification. The class 
of first-order formulas is denoted FO. For a class J 7 of formulas, let co-T be the class of 
formulas —up with ip G F. 
Let J- be a class of formulas. The 

1. boolean closure of J 7 , denoted -B(J r ), 

2. existential first- order closure of J 7 , denoted S^J 7 ), 

3. existential monadic closure of J 7 , denoted ^^"(J 7 ), 

4. first-order closure of J 7 , denoted F0(J 7 ), 

respectively, are defined as the smallest superclass of J 7 that is closed under 

1. boolean combinations, 

2. existential first-order quantifications and positive boolean combinations, 

3. existential set quantifications and positive boolean combinations, 

4. first-order quantifications and boolean combinations, 
respectively. 
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We define E™ n (.F) = J 7 and T^(J=) = Sp n (B(S^ on (J r ))) for every k > 0. Let ng ,on (J r ) = 
co-£™ n (co-.F) for every jfe. We write E™ n and n™ on instead of S^ on (FO) and n™ n (FO), 
respectively. 

The formula classes Y^(J-) and 11° (J 7 ) are defined analogously but for first-order rather 
than set quantification. 

In the sequence, every formula that is equivalent to a formula in J 7 will be called an 
J-'-formula, too. 

A™ on is the class of formulas that are both a £™ on -formula and a n™ on -formula. 

Some of the quantifier alternation classes can be characterized very succinctly by giving 

regular expressions over the alphabet {3,V, a,v} to describe the quantification structure 

of their formulas in prenex normal form, as we did in the introduction. For example, E™ on 

corresponds to 3*V*3*{a, v}*, and n^n™") corresponds to v*a*V*3*{a, v}*. 

The situation is more difficult for classes that involve the boolean closure, the first-order 

closure, or A™ on , as for such a class it is not possible to give a corresponding expression. 

For example, {v, a}*3 {a,v}* corresponds to what is called the positive first-order closure 

o/£™ on in [AFSOOj . and that class is between FO(A™ on ) and FO(S^ on ). 

Example 2.5 In Example 12.41 we have seen that x <2 x' is a £ ^"-formula and also a 
n™ on -formula. Hence it is a A^-formula. □ 

Every boolean combination of A' 1 non -formulas is a A' 1 non -formula. By a standard argument, 
first-order quantification may be expressed by a suitable set quantification in the following 
sense: 

Remark 2.6 Let J 7 D FO is a class of formulas closed under conjunction and <p G J 77 . 
Then there is a formula ip' G J 7 such that 3a; M </? is equivalent to 3X IM (p'. 

By the standard calculation rules of predicate logic and by the above remark, we have the 
following. 

Remark 2.7 Let k > 1 and if be a U° k (A^ on ) -formula. Ifk is odd, then ip is a n£_ 1 (n , 1 non )- 
formula. If k is even, then ip is a n°_ 1 (E' 1 nc ' n ) -formula. Hence in any case p is a n^ on - 
formula. 

2.6 Syntactic Congruence and Syntactic Monoid 

Let L be a word language over alphabet T. The syntactic congruence =i is defined as 
follows: For two words x, y G T* we define x =i y iff for all u, v G T* we have uxv G L <^ 
uyv G L. The syntactic monoid M(L) of L is the quotient of T* by this congruence, i.e., 
M(L) consists of all congruence classes of the syntactic congruence of L. 
A comprehensive introduction to the concept of syntactic monoids of word languages can 
be found in many text books, see e.g. |Pin86j (especially Section 2.4.) or [Str94j (especially 
Theorem V.1.3). 

A group contained in a semigroup S is a subsemigroup of S that is a group. If G is a 
group contained in a monoid M, the neutral element of G may or may not be the same as 
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that of M. A submonoid of a monoid M is a subsemigroup of M that contains the neutral 
element of M. 

Let S, T be semigroups. Then S divides T (written S -< T) iff S is a homomorphic image 
of a subsemigroup of T. Then -< is transitive. 

2.7 Pseudovarieties and Group Complexity 

The group complexity was introduced in |KR65j and assigns a non-negative integer c(S) 
to every semigroup S. We briefly introduce the related notions, see e.g. [Eil76, RS09J. 
A class V of finite semigroups is a pseudovariety (see |RS09j . Definition 1.2.30) if the 
following properties hold: 

• V contains a one-element semigroup; 

• For all semigroups Si, 5*2 G V we have Si x S 2 G V, where Si x S 2 is the direct 
product; 

• For all semigroups S, T with S -< T and T G V we have S G V. 
The following are pseudovarieties: 

• the class G of finite groups, and 

• the class A of finite aperiodic semigroups. 

Definition 2.8 Let S,T be finite semigroups, and let us write S additively. A left action 
of T on S is a map T x S — > S, where the image of (t, s) is denoted ts, that satisfies the 
following two properties: 

• t(s + s') = ts + ts' for every t G T and every s, s' G S, 

• (tt')s = t(t's) for every t,t' G T and every s G S. 

For two semigroups S and T, the semidirect product S x T wrt. a given left action is defined 
as the set S x T with multiplication given by 

(s,t)(s',t') = (s + ts',tt'). 

For two pseudovarieties V and W, the semidirect product V * W is the pseudovariety 
generated by the semigroups of the form V x W with V G V and W G W. 
We define (cf. [RS09] . Definition 4.3.10): 

Co = A, 
C n+ i = A*(G*C n ) 

= C n * (A * G), for every n > 0. 

The famous "prime decomposition theorem" of [KR65] asserts that every finite semigroup 
is in |Jn>i Thus for every finite semigroup S, the group complexity c(S) := min{n | 
S G C n } is well defined. 

For a regular word language L, the group complexity of L is defined as c(L) = c(M(L)). 
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2.8 Height Fragment Technique and Results 

Let V be a class of picture languages over a fixed alphabet Y and let a be a function that 
assigns to every regular word language an element from N. We say that 

• V is at most k-fold exponential wrt. a if for every L G V we have that a(L[m]) is at 
most /c-fold exponential in m. 

• V is at least k-fold exponential wrt. a if there exists L G V such that a(L[m]) is at 
least A;-fold exponential in m. 

The following remark is immediate. Its contraposition provides a technique to separate 
two classes of picture languages, given a complexity measure a as above. 

Remark 2.9 (Height Fragment Technique) LetV,V be picture language classes. 
IfV'QV and V is at most k-fold exponential wrt. a, then so is V . 

We say that a formula class T is at most (or at least, respectively) k-fold exponential wrt. 
a if the class {Mod((p) \ tp G J 7 } is. 

The following is the main result of this paper and will be proved in Section HI 

Theorem 2.10 Let k > 1. The formula classes Ef " (I§_ 1 (Af wn )) , FO(S^ on ), and £™ n 
are both at most and at least k-fold exponential wrt. group complexity. 

By Remark 12.91 this implies. 

Corollary 2.11 Let k > 1. There is a picture language definable by a Y,™°1-formula but 
not by an FO(X£ lon ) -formula. In particular, there is a picture language definable by a 
FO(E%°D -formula but not by an FO(S^ on ) -formula. 

The first statement of the above corollary is a new result. The second statement has 

already been proved in [Mat99j, Corollary 2.31. The proof in this paper is simpler but 

requires at least four (or, with standard encoding arguments, at least two) symbols in the 

alphabet, whereas the proof in [Mat 99] applies also to singleton alphabets. 

Theorem 12.101 also provides new witnesses for the strictness of the alternation hierarchy 

of MSO over pictures, i.e, the result of [Sch97j that is more expressive than S^ on . 

Unlike the witnesses in |Sch971 IMST021 IMat99j . these witnesses are not over a singleton 

alphabet, so the result is formally weaker. 

Another consequence of Theorem 12.101 is the following: 

Corollary 2.12 Let k > 1. There is picture language definable by a £' 1 nc " 1 (FO(S! 1 non ))- 
formula but not by a FO(E^ on ) -formula. 

For the case k — 1, this is the generalization of Corollary 11.21 to picture languages. 



3 EXPRESSIBILITY RESULT 



12 



3 Expressibility Result 

In this section we will prove the upper bound part of Theorem 12.101 For this aim, we 
will construct a picture language whose height-m fragment has group complexity fc-fold 
exponential in m and that is definable in the respective formula classes. 
The idea is to code large Boolean square matrices with pictures of small height. The 
picture language then consists of pictures of the form p\ . . .p n , where pi, . . . ,p n encode 
square matrices whose matrix product over the Boolean semiring is not the zero matrix. 
This way, the syntactic monoid contains the monoid of binary relations, established by 
the syntactic congruence classes of the encoded matrices. By a result of Rhodes (see 
Theorem 13. 1 8[) , that monoid has high group complexity. 



3.1 Iterated Matrix Multiplication 

Let us consider the semiring of mxm-Matrices over the Boolean semiring. Its product ■ is 
the standard matrix product. 

Lemma 3.1 Let m > 1. Then for every n > and all matrices Ai, . . . , A n e {0, i| mxm ; 
we have the following equivalence: 

n 

Y[ A h ^ 3ii, . . .,i n+ i < m : V/i < n : A h (i h ,i h+1 ) = 1. 

h=l 

Proof First, we show by induction on n that the following equivalence holds for every n, 
every k, I e {1, .., n} and all matrices Ay, . . . , A n 6 {0, i} mxm ; 

oiLi a) {k, i) = i 

3ii,..., i n+ i <m:i 1 = kA i n+1 = I A (Wh < n : A h (i h , i h+1 ) = l) . 

This equivalence is immediate for n = 0. Assume that the equivalence holds for some n > 0, 
and let A x ,...,A n+1 G {0, l} mxm . Let k,l < m. Then (IlEMhXM) = ((EK=iA) • 
A n+ ij (k, I). Thus the following equivalence chain holds: 

(nS4)(M> = i 

^ 3l><m:(ir h=1 A h )(k,l') = lAA n+l (l',l) = l 

<^> 3ii, . . . ,i n +i,l' <m:i 1 = k A i n+1 = V A (V/i < n : A h (i h ,i h+1 ) = l) 

AA n+1 (l',l) = 1 
3i u . . .,i n+1 < m : i x = k A (Wh < n : A h (i h ,i h+1 ) = l) 

A A n+1 (i n+ll l) = 1 
3ii, . . . ,i n+2 < m : %\ = k A i n+2 = I A (V/i < n+1 : A h (i h ,i h+1 ) = l). 

This completes the induction. The lemma is now immediate. □ 
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3.2 Picture Languages Defined by a Regular Top- Row Language 

For a non-empty picture p, we define top(p) to be the word of length \p\ in the top row, 
i.e., top(» =p(l,l) ...p(l, 
We observe the following. 

Proposition 3.2 Let T = {0, l} 7 be an attributed alphabet. Let LCT + be a regular word 
language. Then top _1 (L) := {p G | top(p) G L} is definable by a A™ " -formula. 



Proof Let 21 be a deterministic finite automaton that recognizes L. We may assume 
w.l.o.g. that the state set Q of 21 is of the form {0, 1} J for some index set J disjoint from 
I. Let q G Q be the initial state of 21 and F C Q be the set of final states of 21. Let 
ACQxTxQbe the transition relation of 21. 

We will construct a A' 1 non -formula that asserts that the uniquely determined run of 21 on 
the top row of a non-empty input picture is accepting. For this aim, we encode the run 
into an assignment to free variables (X^) Mg j in the obvious way. 
Recall the formulas top(x), left(x), and right(x) from Example 12.31 
For every a : I — > {0, 1}, set 

letter a [x) := A X^x A /\ -^X^x. 

Similarly, for every g : J — > {0, 1}, set 

state q {x) := l\X^x /\ I^^X^x. 



Let 



(/? = Vx(left(x) A top(x) — » state qo (x)) 

^ A( 9 a 9 ')eA VxVx'(staie q (x) A letter a {x) A S^xx' A top[x) — > state q r(x')) . 

Then asserts, for a non-empty input picture p over alphabet {0, 1} /UJ , that top(restrj(p)) 
encodes (in the obvious way) the unique run of 21 on the input word top ( rest 17 (p)). 
Let 

ip = 3x(top(x) A right(x) A \J ^ statef(x)^ . 

Then -0 asserts that for a non-empty input picture over alphabet {0, 1} J that the state 
encoded in the top right corner is final. Consider the formulas 

a = 3(X /i ) M6j (^A^), 
7T = VpT^j^ i/j). 

Both cr and 7r assert for a non-empty picture p over alphabet {0, 1} 1 that the unique run 
of 21 on the top row of p reaches a final state, i.e., that its top row is in L. Thus a and it 
are equivalent and define top _1 (L). This completes the proof. □ 



3 EXPRESSIBILITY RESULT 



14 



Corollary 3.3 Let L C {0, 1} + be a regular language of non-empty words. Let F = {0, l} 1 
be some attributed alphabet, let fi G I be an attribute. Define top~ 1 (L) := {p G 
top(pr At (p)) G L}. There exists a A™ " -formula top-in^L) such that Mod(top-in^(L)) = 
to P/ , '(/-). 

Proof For every picture p G we have the equivalence chain p G top~ 1 (L) iff 

top(pr M (p)) G L iff pr M (top(p)) G L iff p G top _1 (pr^ 1 (L)). Now apply Proposition 13.21 
to the regular word language pr~ 1 (L). □ 

When we write top / ^ 1 (L) in the following, the alphabet T must be clear from the context. 



3.3 Relativization and Closure Under Concatenation 

In this section, we will show the following result: 

Proposition 3.4 For every k > 0, the class of Y^™° n {Jl Q k (/S.™ on ))- definable picture languages 
is closed under column concatenation and column closure. 

Its proof (see end of this section) is well-known for the important case k = 0. We prepare 
the proof for the case k > 1 with the concept of relativization, cf. [Btr94j . Lemma VI. 1.3. 
Recall the definition of the A ^"-formula x <2 x' from Examples 12 A\ I2T51 For a non-empty 
picture p and j, f with 1 < j < f < \p\ we define p[j, f) to be the picture that is assembled 
by the columns j, . . . , f of p. 

Let J, K be disjoint attribute sets. Let ip be a formula with free set variables in {X^ | fi G 
J} and free first-order variables in {x u \ v G K}. Let x, x' be two fresh first-order variables. 
Let ip' be a formula whose free variables are those of ip as well as x,x'. We say that ip' 
relativizes (p to x,x' iff for all non-empty pictures p G Unique JyjK K and all j,f < \p\ we 
have 

P> H V 5 ' ^ i < f an d P[j,f] 1= V 3 - 

Intuitively, <p' says about the subpicture demarcated by the top-row positions x, x' the 
same as p says about the whole picture. 
We will need the following obvious remark: 

Remark 3.5 Let <p\ x ,x'} relativize p to x,x'. Let p\,p2 be two non-empty pictures of the 
same size and j,j' < \p\\ such thatp\[j,j'] =P2[j,j'}- Then 

pi, — — — h ipm ^ P2, — j?- I- iPW 

Lemma 3.6 For ei>en/ first- order formula p that does not use the first-order variables x, x' 
there is a A™ " -formula p>' that relativizes ip to x,x'. 



3 EXPRESSIBILITY RESULT 



15 



Proof Set 

columns = \/xiix 2 (S\Xix 2 — > (X rect xi f-> X rect x 2 )). 

Then columns asserts that X rect is closed under vertical predecessors and successors and 
hence a union of columns. Recall the formulas top(x) and left(x) from Example 12.31 Set 

between = columns A X rcct x A X rcct x' A 
Wxi(S 2 Xix -iX rect a;i) A 
Vx 2 (S 2 x'x 2 -> ^X rcct x 2 ) A 

VxiX 2 (5'2a:iX2 A top(x\) A X\^x' A — )■ (X rect a;2 <->■ X rect a;i)). 

Intuitively, between asserts for a set X rect and two top-row positions x, x' that X rect is 
the subblock cyclically between the columns marked by the top-row positions x and x' . In 
other words, for j, f, m, n > 1 with j, f < n there exists exactly one picture p of size (m, n) 
over alphabet {0, l}{ rcct } such that p, il^iiil [= between. That picture is characterized as 
follows: If j < j', then pr rect (p) carries a 1 exactly in all positions in the columns j, . . . ,f. 
If j > j', then pr rect (p) carries a 1 exactly in all positions in the columns 1, . . . , f, j, . . . , \p\. 
We introduce a formula nowrap that asserts that the first case is true. 

nowrap = left(x) V -<3x 2 (left(x 2 ) A X rect x 2 ). 

For every non-empty picture p over {0, l}^ rect ^ and every < \p\ with p, iMlliii |= 
between we have 

p, ■ |= nowrap j < j'. 

Let (/? be a first-order formula that does not use the variables x, x', and X rect . Assume 
w.l.o.g. that there are no universal quantifications in (p. Let the formula ip' result from ip 
by relativization to X rect , i.e., by successively replacing every first-order quantification of 
the form 3xip by 3x(X rect x A ip). 
Define two formulas a, n as: 

a(x, x') = 3X rect (between A nowrap A <//), 
7r(x, x') = WX rect (between —> (nowrap A (p')). 

Then a G E™ " and n G n™ ". We have for every non-empty picture p and every j,f < \p\ 
that 

Hence ^r X|X '] = a is a A l 1 non -formula and has the desired property. 

This completes the proof. □ 

A formula class T is relativizable iff for every formula p G J 7 and two fresh first-order 
variables x, x', there is a formula tp' G J 7 that relativizes to x, x'. 

Lemma 3.7 For every k > 0, the formula class IL^A™ ") is relativizable. 
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Proof The proof is by induction on k. For the induction basis, let k = and tp G A™ ". 
Then p is equivalent to a formula of the form 3X\ . . . X n (p') as well as to a formula of the 
form VXi . . . X n ((p") for two first-order formulas tp' ; tp". 

As we have shown in the preceding lemma, there exist a £ ^"-formula tp', x n and a n™ "- 

formula tp'/ xx n that relativize tp' (and <p", respectively) to x,x'. 

Let 

a = 3X 1 ...3X n (<p' [x>x , ] ), 

Then cr G SJ 10 " and 7r G IT 1 nc ' n . By Remark 13.51 they are equivalent and relativize tp to x, x' . 
This completes the proof for the case k = 0. 

Now assume k > 1 and the claim is true for k — 1 instead of k. Let c/2 G 11° (A™ "). Then 
<^ is equivalent to a formula of the form 

Vxi . . .Vx n (-i^), 

for some n > 1 and ^ £ n°_ 1 (A! ] non ). By assumption there is a n°_ 1 (A5 non )-formula V'fe.a:'] 
that relativizes tp to x, x'. Choose 

<fi[x,x'} = Vxi . . . Vx n ^ X <2 Xi A Xi < 2 x'^ -)■ ->1p[ x>x r 

Since x <2 x' is a A l 1 non -formula, ^[a;,^'] is indeed a n°(A™ on )-formula. Besides, v 9 ^^'] 
relativizes tp to x, x'. This completes the proof. □ 

In Section 14.31 we will come back to the following observation. See also Remark 12.21 

Remark 3.8 Let J 7 be a class of formulas closed under conjunction and disjunction. The 
class of £5"°" (J 7 )- definable picture languages is closed under intersection, union, and al- 
phabet projection. 

Lemma 3.9 Let T D A™ on be a relativizable class of formulas. Let T = {0,1} 7 . Let 
Li,L 2 C be two E™ " (J 7 ) -definable picture languages. 

1. If J 7 is closed under conjunction and disjunction, then L\L 2 is T,™ on '(J 7 ) -definable. 

2. If J 7 is closed under conjunction, disjunction, and universal FO-quantification, then 
L\ is H™ n {T) -definable. 



Proof of Lemma \5M Claim [JJ Let tp,ip G ^"(J 7 ) be formulas with Mod(p) = L x 
and Mod(ip) = L 2 . Let xo,Xi,X2,X3 be four fresh first-order variables. Choose formulas 
(^[ l0]Xl ] and i/j[x 2 ,x 3 ] that relativize <p and if), respectively. Recall the formulas top(x), left(x), 
and right(x) from Example 12.31 Choose 

tp' = 3xo, Xi, x 2 , Xz(top(xo) A top(xi) A top(x 2 ) A top(xs) A 

left(x ) A S 2 xix 2 A right(x 3 ) A tp [x0;Xl] A ip[ X2 , X3 }) ■ 
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Then iff G E?(E5 non (J r )) = E™"^). It remains to show that Mod{<p') = L X L 2 . 
Let p G Mod(p'). Then there exist top-row positions (1, Jo), (1, Ji), (1, J2), (1, J3) G dom(p) 
such that j Q = 1 and j x + 1 = j 2 and j 3 = |p| and p[j , ji] (= and p[? 2 , j 3 ] |= V>- This 
implies p G Mod(p)Mod(i/j) = LiL 2 . Thus we have shown Mod(ip') C L1L2. The converse 
inclusion is similar. This completes the proof of Claim [TJ □ 



Proof of Lemma 13. 9|, Claim [2] Let X = (X^^j be a tuple of set variables and ip E J 7 
be a formula such that 

Mod(3Xip) = L x . 

Let end be a fresh attribute. Choose A^-formulas top-in end (0*l) and top-in cnd ({0 , 1}*1) 
according to Corollary 13.31 Set <p' = ip A fop-m end (0*l). Let x, y be fresh first-order 
variables. Let yLyi be an ^-formula that relativizes ip' to x,y. Let no-between be a 
A J 10 " -formula that relativizes fop-m end (0*l) to x, y. 
Recall formula top(x) from Example 12.31 Set 

next end = top(x) A top(y) A (Vw(S , 2 wx — >■ X end «;)) A no-between. 
if" = \JxVy(next end ip' [xy] ). 

<p'" = 3X3X cnd (if" Atop-in CDd ({0,l}*l)). 
Then next end is a A ^"-formula and </' G E^T"). We claim that 

Mod(p"') = Lf. (1) 

Let J' = IU JU {end}. 

First let p G Modi{<p"'). There exists a picture p" G Modr(p>") such that expend} (p") = p 
andtop(pr end (p"))G{0,l}*l. 

Pick n > 1 and top-row positions (1, ji), . . . , (1, j n ) such that ji < ... < j n such that 
these positions are those top-row positions of p" that carry 1 for attribute end. (Note that 
jn = H-) Choose jo = 0. 

Choose a decomposition p" = p[ . . . p' n into non-empty pictures p[, . . . ,p' n such that \p[ . . . p' k \ 
j k for every k G {1, ..,n}. For every k, choose picture p k over alphabet {0, l} 7 such that 
ex JU{cnd} (p' k ) =p k . Then 

p = p 1 ...p n . 

Let k G {l,..,n}. The position (1, jfe—i + 1) is a position in the picture p" whose S 2 - 
predecessor either does not exist (in case k = 1) or carries a 1 for attribute end. Thus 



Since p" \= ip", this implies 



_// <i,jfc-i+i) i_ / 

P ' x F~ F= V[x,y}i 



which in turn implies p' k G Modp(<p') C Mod//(<£>) = Mod/ujujendj (y>) . Thus ex{ end }(p / A .) G 
Modiuj(tp). By choice of p^, we have p k G Modj(3X</?) = L x . 
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Since k has been chosen arbitrarily from {1, ..,n}, this implies p = p\ . . .p n G L™ C LJ. 
This completes the proof of the direction "C" of Equation 

For the converse direction, let p G L\. Pick n > 1 and pi, . . . ,p n G Li such that p = 
Pi . . . p n . Choose jo = and for k G {1, ..,n}, choose jk — |pi • • -Pfc|. 

For every fc, we have p& G Mod/(3X(<^)), thus for every k, there exists p' k G Modi'((p) such 
that exju{end}(Pfc) = Pit- Furthermore, we may pick p' k in such a way that top(pr end (j4)) G 
0*1. 

Choose p" = p' l .. .p' n . Then ex JU{end} (p") = p and top(pr cnd (p")) G (0*1)- C {0, 1}*1. In 
order to show p G Modi((f'"), it remains to show p" G Modp((p"). To see this, let 0, z' 
be positions in p" such that p",^j \= next ead (x,y). By definition of next end , these are 
top-row positions, so there are j, f such that z = and z' = and (because 

top(pr end (p' fc )) G 0*1 for every k), the columns j and j' are the start- and end-column of 
one and the same p' fc -subblock of p" (for some k G {1, -,,n}). Since p' k \= tpA top-in end (0*l), 
we conclude that this p' fc -subblock is in Mod(ip') , i.e., p", ~ |= V'jx y]- Since z, 2' have been 
chosen arbitrarily, this implies that p" |= ip". We also have |= top-in end ({0, 1}*1). This 
shows p |= ip'" and completes the proof of Equation [T] and of Lemma 13.91 □ 

We are now ready to prove the result of this section. 

Proof of Proposition 13.41 For k > 1, this is a consequence of Lemmas 13.71 13.91 and 

the fact that n^AJ 10 ") is closed under conjunction, disjunction, and universal first-order 
quantifications. 

For k = 0, we have S5 non (n°(A'j non )) = SJ 10 ", which is the class of recognizable picture 
languages. For this class, the statement is well known, see e.g. |GRST96] . □ 

3.4 Assembling the Picture Language 

For the rest of this section, let J 7 D A™°" be a class of formulas such that the class of 
S5 non (J r ) -definable picture languages is closed under column concatenation, column clo- 
sure, union and intersection. We will need our results only for the cases provided by 
Proposition I3.4[ i.e., for the case J 7 = n°(A l 1 non ) for some k > 0. 

For a function / : N>i — > N>i and an alphabet T, the picture language associated to f is 
defined as 

L /ir = {p G r + <+ I \p\ = f(p)}. 

Typically T is clear from the context or irrelevant because of Remark 12.11 so we usually 
omit the second subscript. 

If / : N>i N>i is a function, we define another function / + 1 : N>i — > N>i, n y f(n) + l. 

Remark 3.10 Let us denote the set of all non-empty pictures of length 1 over T by r +>1 . 
Then £/+i,r = LfpT + ' 1 . Clearly r +>1 is J 7 -definable, thus if Lj is £™ on (^-definable, then, 
by Lemma WT^, so is Lf +1 . 
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If n > 1 and i,j G {1, -.,n}, set code n (i,j) — (i — l)n + j. Then code n defines a bijection 
from {1, .., n} x {1, .., n} onto {1, .., n 2 }. 

If tt. > 1 and p is a picture of length n 2 over {0, 1}, we define fold n (p) as the nxn- matrix 
over {0, 1} with fold n (p)(i, j) = p(l, code n {i,j)) for every i, j G {1, .., n}. 
Every top-row position of a picture p of length n 2 corresponds to exactly one position in 
the n x n- matrix fold n (p). The intuition for the attributes used in the following lemma is: 
Picture positions marked by blk (or diag, or end) correspond to the matrix positions in 
the right column (or on the diagonal, or in the bottom-right corner, respectively). 

Lemma 3.11 Let f : N>i — > N>i be a function such that its associated picture language 
Lf is E5 non (J r )- definable. Consider attribute set I = {diag, end, blk}. 
There exists a E™ "^)- definable picture language L such that for all pictures p over 
alphabet {0, l} 7 we have: p G L iff \p\ = f(p) 2 and for every k < \p\ we have the following 
three conditions: 

P (i,k)(bik) = i m\k, (2) 

p(l,k)(diag) = l & (/@ + l)|(fc-l), (3) 
p(l,k)(end) = 1 k = f(p) 2 . (4) 

Proof First, we observe that for every k < \p\ the Equivalence ([3]) is equivalent to 

p(l,k)(diag) = l^ke {code m (i,i) | % G {1, .., f(p)}}. (5) 

This is because for every k < \p\, we have the equivalence chain: (f(p) + 1) | (k — 1) iff 
3a>0 : a(f(p) + 1) = k - 1 iff 3a>0 : af(p)+a+l = A; iff 3a>0 : code^a + 1, a + 1) = fc 
iff 3i>l : codef(-p)(i,i) = k. 

Recall the definition of top^ 1 from Corollary 13.31 Let 

Mx = ^ntop^on), 

M 2 = L /+1 ntop^ g (0*l), 
^3 = top^ ag (l)M 2 + , 

m 4 = M+nM 3 n top-yon), 

X = {ae {0, l}' I a(diag) = 1 A a(blk) = 1 a(end) = 1}, 
M 5 = top" 1 ^), 
L = M 4 n M 5 . 

Then L is SJ 10 " (^-definable by Corollary EES Remarks ES EES and Lemma El 
For the only-if-direction of the lemma, let p G Lq. Since p G M] 4 ", it may be decomposed into 
subpictures p — p\ . . .p n such that for each h < n we have \ph\ = f(p) and top(pr blk (p h )) G 
0*1. This implies (T5]) for every k < |p|. 

Since p G M 3 , it may be decomposed into subpictures p = qp[ . . .p' n , such that n' > 1 
and |g| = 1 and top(pr diag (g)) = 1 and for every h < n' we have |pJJ = /(p) + 1 and 
top(pr diag (ph)) G 0*1. Thus for every k < \p\ we have Equivalence (j3J) and hence ([5j). 
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Since p G top^^O+l), the position (1, |p|) is the only top-row position that carries a 1 for 
attribute end, thus (by p G M 5 ), the value \p\ is the only value for k that fulfills the right 
sides of both (T2|) and OH]) , thus 

\p\ = min{fc > 1 | f(p) | k A (/(p) + 1) | (k - 1)}. 

Thus \p\ = f(p) 2 and (0J. This completes the proof of the only-if-direction. 
For the converse direction, let p be a picture over {0, 1} / such that \p\ = f(j>) 2 and Equiv- 
alences OS])-® hold for every k < \p\. Equivalences ©-(H]) and \p\ = f(p) 2 imply p G M 5 . 
Equivalence (jlj) and \p\ = f(p) 2 imply top(pr end (p)) G 0*1. From ([2]) we conclude p G M+ . 
From ([3]) we conclude p G M 3 . This implies p G M x + n M 3 n top^On) = M 4 . This 
completes the proof of the lemma. □ 

The intuition that the top-row positions of a picture correspond to square matrix positions 
is helpful to understand the next lemma, too. Here, the attributes intuitively mean: The 
attribute inp marks those picture positions where the Boolean "input" matrix carries a 
one. The attribute piv picks one of those as the "pivot" . The attributes row (and col) 
mark the position on the diagonal that shares the row (or column, respectively) with the 
pivot position. 

Lemma 3.12 Let f : N>i — > N>i be a function such that its associated picture language 
Lf is E™ " (J 7 ) -definable. There exists a E™ on (J 7 ) -definable picture language L 7 such that 
for all pictures p over alphabet {0, 1 }{ jn P>p jv i row > coi i end } we have: p G L 7 iff \p\ = f(p) 2 and 
there are i,j < f(p) such that for all k < \p\ we have 

p(l,k)(end) = 1 k = f(p) 2 , (6) 

p(l,k)(piv) = l p{l,k)(inp) = l, (7) 

p(l,k)(piv) = 1 k = code f j), (8) 

p(l, k)(row) = 1 <^ k = code/(p)(z, i), (9) 

p(l,k)(coL) = 1 <£> k = code m (j,j). (10) 

Proof Recall the definition of from Remark 13.101 Choose Lq over the alphabet 
{0, 1 }i dia 9, cnd , blk } according to Lemma \3. Hi We define the following picture languages over 

alphabet T = {0, l}{inp, P iv,row,col,end,diag,blk} 

"^1 iriD.Div.row.col\ ("^C 



L 2 = top- 1 v (0*)(top^(l)ntop- 1 v (l))to Ppi 1 v (0* 



L 3 = top-U0*)(top^ ag (l)ntop- 1 w (l))top- 1 w (0*), 
U = top^(0*)(top- 1 ag (l)ntop- o Kl))top^(0*), 

n 5 = top^on u o*) n ((to Pro 1 w (io*) n top-^(on)) u 

(top^io^ntop-Uon))), 

L 5 = T*'*N 5 T*'*, 

n 6 = (L;r+' 1 )n((top- 1 J (io*)nto P ; i 1 v (o*i))u 

(top^(io*)nto P -i(o*i))), 

l 6 = r*-*iv 6 r*'*. 
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Finally, we define the following picture language over alphabet {0, 1 }{ 1D P<P lv < row > coI > cnd } : 

L>7 = ex{diag,bifc}(£i n . . . n L 6 ). 

By Corollary I3.3[ Remark 13. 8[ and Lemma [3. 9 \ L-j is ^^"(J-^-definable. 
For the only-if-direction, let p G L-j. For abbreviation, set n = /(p). Pick picture p over V 
such that p G L\ D . . . D L§ and ex{di a g,wjc}(p) = P- We have to show the claim about the 
size of p as well as the Implications (ItjjM llOp for every k < \p\. 

The choice of L\ (or rather, of L ) implies \p\ = n 2 as well as (j6]). 
Implication follows from p G L 2 . 
Since p G L 2 , there exists /c pjv such that 

Wk : p(l, k)(piv) — 1 ^ A; p ; v = /c. 

Choose i,j<n such that & pjv = code n (i,j). This ensures Equivalence (jHJ). 
Similarly, since p E L 3 , there exist A; row such that 

V/c : p(l, fc)(row) = 1 A; row = A;. 

Choose i row , j row < n such that 

krow COd6 n (i raW) jrow) ■ 

Since p G L 5 , the substring of top(p) demarcated by k piv and fc row is in the same of its n 
blocks of length n, thus we have i = i row . Since p G L 3 and fl5J), we have i row = j row . Thus 
^row = code n (i,i), which proves Equivalence (j5J). 
Similarly, since p G L4, there exist & co j such that 

V/c : p(l, k)(col) = 1 <^> /c coi = fc. 

Choose icohjcoi < ft such that 

k co i = code 

n \^coU J col ) ■ 

Since p G L 6 , the difference |& co j — & pi v| is a multiple of n, thus j = j co j. Since p G L 4 
and (jSJ), we have z co j = j co \. Thus fc co j = code n (j,j), which proves Equivalence (fTUI) . This 
completes the proof of the only-if-direction of this lemma. 

For the converse direction, let p be a picture over alphabet {0, 1 }{ in P>P iv > row > ccJ > end } such that 
|p| = f{p) 2 , and let i,j < /(p) such that for all k < \p\ we have Implications (l6l)-lJT0l). 
Again, we set n = /(p) for abbreviation. Choose picture p over alphabet T such that 
restr{i nP)PJ v roWiCO j ienc ;} (p) = p and Equivalences (j2D and from Lemma 13.111 hold for 
restr {diagiendjbiJf }(p) instead of p. 
We have to verify that p G Li fl . . . fl L 6 . 

Ad L x : The picture restr {diagiendj5iJc} (p) fulfills LemmaEHU ©-(HI), so restr {dia&end)bJi .}(p) G 
L , thus p G restr^ iag)end)bJi} (L ) = ex^ ppiV)roW)CoJ} (L ) = L x . 
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Ad L 2 : Choose a decomposition p = qrs with \q\ = code n (i,j) — 1 and \r\ = 1. By (JHJ) we 

have 1 = p(l, code n (i, j))(piv) = r(l,l)(piv). From ((Tj) it follows 1 = r(l,l)(inp). Thus 

r G top~ iv (l) fl top^p(l). Again by (jE}, for all A; < code n (i,j) we have = p(l, k)(piv) = 

q(l, fc)(piv), thus q E topj£(0*). Similarly s E top^(0*). 

We have shown p = qrs E top^(0*)(top^(l) D topr^(l))top pi 1 v (0*) = L 2 . 

Ad L 3 : Similar to L 2 , but apply (jSJ) instead of ©, and © instead of (jHJ). 

Ad L 4 : Similar to L 2 , but apply (jSJ) instead of ([7]), and ffTUl) instead of (jSJ). 

Ad L 5 : Choose a decomposition p = qrs with |g| = min{code n (z, i), code n (i, j)} — 1 and 
\r\ = \code n (i,i) — code n (i,j) \ + 1. To complete the proof that p E L 5 , it suffices to show 
that r E N 5 . 

In case % < j, the picture r is the infix of p at the columns code n (i , i) , . . . , code n (i , j) 
(inclusively). None of these numbers (except for maybe the last one) is a multiple of n, thus 
(by (0)) none of the corresponding top-row positions carries a 1 for attribute blk, thus r E 
top^ JJc (0*l U 0*). Furthermore, r(l, code n (i, i))(row) = 1 by (jHJ) and r(l, code n (i, j))(piv) = 
1 by (|H1), which implies r E top7 o 1 w (10*) PI top~y(0*l). We have shown r E N 5 . 
In case % > j, one similarly shows r G top^(0*l U 0*) and r G top~y(10*) fl top~^(0*l), 
which also implies r E N§. 

Ad L 6 : Choose a decomposition p = qrs with |g| = min{code n (j, j), code n (i, j)} — 1 and 
|r| = |code n (j,j) — code n (i, j) \ + 1. To complete the proof that p E L 6 , it suffices to show 
that r G iV 6 . We have that \code n (j,j) — code n (i,j)\ is a multiple of n, thus \r\ — 1 is a 
multiple of n = f(p), hence r G L^T + ' 1 . 

In case z < j, one shows r G top pi y(10*) fl top^O*!), which implies r G iV 6 . 

In case i > j, one shows r G top~ o j(10*) fl top pi y(0*l), which also implies r G iV 6 . 

We have shown p E L\ H . . . fl L§, which completes the proof of the lemma. □ 

In the next lemma, we consider a concatenations of a sequence of pictures, each of which 
encodes a Boolean "input" square matrix in its top row. The marker attributes introduced 
in the preceding lemmas help to calculate one position of their matrix multiplication of 
these input matrices. 

Lemma 3.13 Consider the alphabet V = {0, i}i in P< cnd } . Let f : N>i -»- f% be a function 
such that its associated picture language Lf is ^^"(J 7 )- definable. Let L be the picture 
language overT containing all pictures of the form p\ . . .p n where n > 1 andpi, . . . ,p n have 
length f{p) 2 andtop(pr cnd (p h )) E 0*1 (for all hE {l,..,n}) and J]Li fold /(p)(P r i np (^)) + 0, 
where the product refers to standard matrix multiplication over the Boolean semiring. 
Then L is E™ n {F)- definable. 

Proof Set S = {0, i}{™p,p™,row,coi,end} Le t L 7 be the picture language over £ defined in 
Lemma \3. 121 Let denote the set of pictures of length 1 over S. Consider the following 
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picture languages over S: 

Q = topi((ri), 

M = L*S+- 1 = {p G | (|p|-l) is a multiple of f(p)}, 

n = topi(i{o,i}*)ntop^ r ({o,i}*i)ntopi(ono+(iue))nM, 

L 8 = (s*'*A^s*'*) n (QQ), 

L 9 = (L 8 +Q)UL 8 + , 
L' g = (QL g ) U Q, 

L w = (L7 n L 9 n L' g ) u L 7 . 

Let L\i = ex{ pJ v jroW)CO i}(Lio). Then L\\ is S^ " (^-definable by Corollary 13.31 Remark I3~8l 
Lemma [3.91 and Lemma [3. 121 We claim that L\\ = L. 

First, let p G L n . We will show that p G L. There exists p G L 10 such that p = 
e x {piV,row,coJ}(p)- Since p G L7 , there exist n > 1 and pi, . . . ,p n G I/7 such that p = p\ . . .p n . 
Choose pictures pi, . . . ,p n over T such that for every h < n we have = ex^^o^o]}^). 
Then p = pi . . .p n . Since (by (jH])) we have L 7 C Q, we obtain top(pr end (p/ l )) G 0*1 and 
thus top(pr cnd (p /i )) G 0*1, as required. 

As a shorthand, we set m = p. By Lemma |3.12[ we have \ph\ = f{m) 2 for every h < n. 
Choose ii, . . . , i n ,ji, ■ ■ ■ ,jn < fi^n) 2 according to Lemma 13.121 i.e., such that for every 
h < n and every k < f(p) 2 , we have 

p h (l,k)(piv) = l => p(l,k)(inp) = l, (11) 

p h {l, k)(piv) = 1 <^> k = code f(m) (i h ,j h ), (12) 

A;)(row) = 1 <=> k = code f ( m )(i h ,i h ), (13) 

p h {l,k)(col) = 1 <^> k = code f{rn) (j hl j h ). (14) 

Furthermore, choose z n+ i = j n . We claim that 

= Jft for every ft < n. (15) 

To see this, let h < n. If h = n, we have z/i+i = by choice of i n +i- So we may assume 
h < n and n > 1. We note that phPh+i £ £*'*iV£*'*. (If /1 is odd, this follows from j3 G Lg; 
if /i is even, this follows from p G L' g .) This means that phph+i has an infix r G N. Let 
^ ^ < 2/(m) 2 be the start and end positions of that infix r in phph+i- Since ph,Ph+i G £7, 
we have = |pVt-i| = f(rn) 2 . Since ph,ph+i G Q, the only two top row positions of 
PhPh+i carrying a 1 for attribute end are the rightmost positions of ph and of ph+i- Since 
r G top m 1 d (0*10 + (l U e)), the infix r overlaps the center of phPh+i, i-e., / < f{jn) 2 and 
V > f(m) 2 . Since r G iV, we have 

1 = r(l,l)(col) = (^ +1 )(l,/)(col)=^(l,0(col) (16) 
1 = r(l,\r\)(row) = (p h p h+1 )(l,l')(row) = p h+1 (l,l' - f(m) 2 )(row) (17) 



By dUD and (USD, we have 

/ = code f{m) (j h ,j h ) = (j h - l)f(m) 2 + j h . 
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By ([I3D and (HZD, we have 



V - f{mf 



code f ( m )(ih+i,ih+i) = (ih+i 



l)f(m) 2 + i h+1 . 



(19) 



Since r G M we have 



/(m) | /' - I = i h+1 f(m) 2 + i h+x - (j h 



l)f(m) 2 -j h . 



(20) 



We may conclude from (1201) that f(m) \ ih+i — jh and hence ih+i = jh- This completes the 
proof of ( TT5|) . 

For every /i G {1, ..,n}, choose A h = fold m (pr inp (p h )). 

Let h < n. By flTJ), we have p h {l,code f ( m )(i h ,j h ))(piv) = 1. Thus A h {i h ,i h+1 ) = 
A h (ihJh) = Ph(l,code f ( m) (i h ,j h ))(inp) = 1. (For the last equality, see (HI]).) By Lem- 
ma [3J], this implies n^=i 4^0 and hence p = eX{ p j V)IW!CC ,j} (p) G L. This completes the 
proof of L u C L. 

For the converse direction, let p <E L. Again, set m — p. Choose n > 1 and pictures 
Pi, . . . ,p n of length f{m) 2 over Y such that p = p\ . . .p n and top(pr cnd (p/ l )) G 0*1 (for all 
h G {1, .,«}) and nLi fold /(m)(pri np (Pft)) ^ 0. 

We claim that there exists a picture p G with ex{ piViroWiCO j}(p) = p and p G Li . By 
Lemma |3TT| there exist ii, . . . ,i n < f(m) such that fold m (pr inp (pft))(z/j, ih+i) = 1 for every 
h < n. For every h < n, choose ph as follows: 



(The letters of ph at positions (i,k) with i ^ 1 are irrelevant and may be set arbitrarily.) 
It is straightforward to check that ph G L-j for every h < n. We set p = p\ . . .p n . We have 



If n = 1, we have p G L 7 C L 10 . Thus for completing the proof that p G L 10 , we may 
assume n > 2 and show that p G L 9 D L' 9 . 

By definition of Q an d L$, for showing p G Lg it suffices to show 



for every odd h < n, whereas for p G Lg it suffices to show (j2ip for every even h < n. So 
for completing the proof that p G Lio it suffices to show (12 ip for every h < n. 
Let h < n. Let I = codef^iih+i^h+i) an d V = f{m) 2 + L Then the infix of phPh+i from 
column I to £' (inclusively) is indeed in N. This completes the proof that p G L w and thus 



restr {inp>cnd} (p ft ) = p h , 

p h {l,k)(piv) = 1 O k = code f(m) (i h ,i h+1 ), 

p h (l,k)(row) = 1 <^ k = code f ( m )(i h ,i h ), 

p h (l, k)(col) = 1 k = code f(m) (i h+1 , i h+1 ). 



p G U 



p h p h+l G S*'*iVS 



(21) 



the proof of the lemma. 



□ 



Definition 3.14 (Monoid of binary relations) Let n > 1. TTie monoid B n is the set 
of binary relations over {1, .., n} together with the usual relation product. 
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This monoid of binary relations has been studied extensively. We will use it in the lower 
bound proof (see Section @|. Besides, we use it to state the next lemma a little more 
general than needed. 

Lemma 3.15 Consider the alphabet T = {0, l}i in P< cnd } . Let f : N>i -> N>i be a function 
such that its associated picture language Lf is E™ " (J 7 ) -definable. There is a S5 non (J r )- 
definable picture language L over F such that for every m > 1 the syntactic monoid of 
L[m] contains a submonoid isomorphic to -B/( m )- 

Proof Let L be defined according to Lemma [3. 131 Then L is indeed E5 non (J r )-definable. 
Let m > 1. Set n = f(m). We claim that the syntactic monoid M(L[m}) of the height-m 
fragment of L contains a submonoid isomorphic to B n . If n = 1, this is trivial, so we may 
assume n > 2. 

For a relation 7r G £>„, its characteristic matrix is the nxn-matrix defined by 



For a relation 7r £ B n , we define as the picture of size (m, n 2 ) over T such that the top 
row of p n is defined as follows: top(pr end (p 7r )) G 0*1 and fold n (pr inp (p 7r )) = A n . The letters 
of p w at positions with i ^ 1 are irrelevant and may be chosen arbitrarily. 
For a non-empty picture p of height m, let [p] denote its congruence class wrt. the syntactic 
congruence of L[m]. We claim that the mapping ip : n i— >• is an injective homomorphism 
from B n into the syntactic semigroup of L[m}. 

First we show that ip is a homomorphism. For this aim, we have to show that for every 
7T, r G _B n , the pictures p^. T and PTrPr are syntactically congruent wrt. L[m]. So let 7r, r G 5 n . 
Let g,r G r m '*. We will show that 



Assume qp^rr G L. Each picture of L is a column concatenation of one or more blocks of 
size (m,n 2 ), each of which is in 1x^^(0*1), i.e., demarcated by a 1 for attribute end in its 
upper right corner. Since qp WT r G L and p nT is one of these blocks, the pictures q and r are 
assembled by zero or more of these blocks. In other words, there are s, s' > and pictures 
qx, . . . , q s , ri, . . . , r s i of size (m, n 2 ) such that 




qp-n T r eio qp-Ep T r G L 



(22) 



q = qi . . . q s and r = n . . . r s i . 



Let • denote standard matrix multiplication over the Boolean semiring. Choose 



B 

C 



fold n (pr inp (gi)) • • •fold n (pr inp (&)), 
fold n (pr i (n)) • ■ ■fold n (pr in (r,0)- 



Since qp^r G L and by 



fold n (pr i n Jp VT )) 



A 



-TIT 



A 7T -A T = fold n (pr inp (^)) • fold n (pr in (p T )) 
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we have 



^ B ■ fold n (pr inp (jv T )) -C = B- fold n (pr inp (p 7r )) • fold„(pr inp (p r )) • C, 



which implies qp n p T r G L. This completes the proof of the direction "=>■" of (1221) . The other 
direction is similar. This completes the proof that ip is a homomorphism, so it remains to 
show that tp is injective. 

For every a,b G {1, .., n} we define the square matrix C a ^ G {0, l} nxn by 

C a ,b(i,j) = l<^ = (a,b). 
Recall that n > 2. Then for every 7r e 5„ we have 



{(l,6)M(o.2)} 



{(1,2)} if(6,a)G7r 
otherwise, 



hence for the characteristic matrix A we have 

<V*.^-{^ ,23) 

where denotes the zero matrix of size nxn. 
Now let 7r, 7r' G _B„ such that = [pv]- Then 

Vg, r G T m ' + : gp„-r G L[m] ^ <$V?" G L[m}. 

This implies (for g, r with C li6 = fold n (pr inp (g)) and C ai2 = fold n (pr inp (r))): 

Vet, b G {1, .., n} : C x ,b • A • Ca, 2 7^ ^ C li6 • A y ■ C a , 2 ^ 0, 

which by (123]) implies 

Va, 6 G {1, .., n} : (b, a) £ n <^> (6, a) ^ 7r'. 

This implies n = tx' . We have shown that <p is injective, which completes the proof. □ 
We need the following result: 

Theorem 3.16 ([Sch97| IMat9 9]) Let k > 1. There is a k-fold exponential function f 
such that Lf (over a singleton alphabet) is S? lon (ng_ 1 (A? lon ))- definable. 

The formula construction for the above proof is in |Sch97] . except for the observation 
that the inner quantifier block may be chosen as universal as well as as existential. For a 
construction including this observation, see |Mat99j . Theorem 2.29. 

Lemma 3.17 Let k > 1. There is a k-fold exponential function f and a £™ on (rT)!_ 1 (A' 1 nc ' n ))- 
definable picture language L over alphabet {0, l}{ m P> enc1 } such that for every m > 1, the 
syntactic monoid of L[m] contains a submonoid isomorphic to -B/( m ). 
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Proof By the preceding theorem, there is a A;- fold exponential function / such that Lf 
is definable in the specified formula class. The class of picture languages definable in 
n°_ 1 (A l 1 non ) is closed under column concatenation and column closure by Proposition I3.4L 
The claim follows from Lemma 13.151 □ 

For the following theorem from }Rho74a] see also the remarks in [RS09j . page 307. 
Theorem 3.18 ( [ Rho74a] ) c(B n ) = n — 1 for every n > 1. 
Lemma 13.171 and Theorem 13.181 imply 

Theorem 3.19 For every k > 1, the class of E' 1 non (n° ^A™ ")) -definable picture lan- 
guages over {0, l}{ jn P< cnd } is at least k-fold exponential wrt. group complexity. 

This is the upper bound part of Theorem 12.101 and the result of this section. 

4 Non-Expressibility Result 

4.1 Some More Notation 

In this section, we write a (monoid or semigroup) homomorphism to the right of its argu- 
ment. Consequently, if rj : M — > N is a homomorphism, the image of M under r\ is denoted 
Mi], and the pre- image of a subset X of N is denoted Xi] -1 . If n : N — > T is another 
homomorphism, then rjn denotes the composition of r\ and 7r. 

We recall some notions in addition to those in Section 12.61 Let L be a word language over 
T. If M is a monoid and r] : T* — > M is a homomorphism such that there exists a subset 
X C M with L = Xr]' 1 , then we say that M and rj recognize L. We are only interested in 
the case that M is finite. 

The syntactic homomorphism t]l ■ T* — >■ M(L) maps every word to its syntactic congruence 
class. M(L) and tjl recognize L. Besides, if 77 is a homomorphism that recognizes L, then 
tjl factors through 77, meaning that there exists a homomorphism 9 : T*r] — > M(L) such 
that i]8 = r) L . 

4.2 Transition Monoid of an NFA 

Let L be recognized by a non-deterministic finite automaton (NFA) 21 with c states. Using 
the transition structure of L, one can construct the transition monoid M% of 21, which is 
a monoid with 2 C elements that recognizes L. We sketch this construction from [Pin96j . 
Let Q be the state set of 21 and let T be its alphabet. Let M% = {0, 1}Q X( 2. M% forms a 
monoid, with the multiplication given as follows: for all q, q" G Q we set 

(A.B)(q iq ") = J2Mq,q')-B(q',q"), 

q'eQ 
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where sum and product on the right refer to the Boolean semiring. To every letter a G T, 
we assign the element 5 a G M<& such that for every q,q' G Q we have S a (q, q') = 1 iff there 
is a transition from q to q' labeled a. 

The mapping 5 : a t— )■ 5 a induces a monoid homomorphism T* — > M%. Then L is the 
pre-image of {(i, f)eQxQ\iis initial and / is final} under 5, thus Mj and 6 recognize 
L. 

4.3 A Semantic Equivalent to First-Order Quantification 

In this section, let / be an attribute set, K C J, and V = {0, l} 7 . Recall from Remark 12.21 
how the alphabet projection ex{ M } corresponds to the set quantification over variable X^. 
Similarly, the syntactic concepts of disjunction and negation correspond to union and 
complementation wrt. Unique IK , respectively, in the sense that 

Modi, K (<P V VO = Mod IiK (tp) U Mod IjK {^): 
Modi t K(^(p) = Unique j j^Modi^K^)- 

The next definition and remark present an operation on picture languages that similarly 
corresponds to the effect of first-order quantification. 

Definition 4.1 Let u. G /. Let L be a picture language over alphabet {0, l} 1 . Then 
exfo At (L) denotes the set of non-empty pictures p over alphabet {0, ljAM j or which there 
exists a picture p' G Uniquej ^ fl L such that p = ex^{p'). 

Remark 4.2 Let fj, G K , let ip be a formula, and let be a first-order variable. 

Mod A{/l}iJ K\ {/ 4(3x M y?) = exio^(Mod I>K (tp)). 

4.4 The Block Product 

The block product has been introduced in |RT89j . That block product captures the effect 
of first-order quantification on the syntactic monoid in the sense of Lemma 14.31 below. 
Following the presentation of [Str94j . pp. 61-65, we prepare the definition of the block 
product by introducing bilateral semidirect products. 

Let S, T be finite monoids, and let us write S additively. Assume a given left action as in 
Definition 12.81 The action is monoidal if it additionally satisfies: 

• Is = s for every s G S, 

• tO = for every t G T. 

Monoidal right actions are defined dually. A left and a right action of T on S are compatible 
if (ts)t' = t(st') for every t,t' G T and every s G S. 

Given a pair of compatible actions of S on T, we define the bilateral semidirect product 
S ** T. This is the set S x T with multiplication given by 



(s,t)(s',t') = (st' + ts',tt'). 
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If M and N are monoids and the underlying left and right actions are monoidal, then the 
bilateral semidirect product M ** N is indeed a monoid, see |Str94j . Proposition V.4.1. 
We remark that by [Str94], Example V.4.b, the semidirect product from Section \2.7\ is a 
special case of the bilateral semidirect product with the right action defined by st = s. 
Closely following [Str94], we can now define the block product. Let M, N be monoids, but 
this time we will write the products in both of these monoids multiplicatively. The set 
M NxN of all maps from N x N into M forms a monoid under the component- wise product, 
which we write additively. That is, for Fx, F 2 : N x iV — > M we define F = Fi + F 2 , where 
F(ni,n 2 ) = F 1 (n 1 ,n 2 ) ■ F 2 (ni,n 2 ) for all n\,n 2 G N. Thus M NxN is isomorphic to the 
direct product of |iV| 2 copies of M. The identity of this monoid is the map that sends 
every element of N X N to 1. We define left and right actions of iV on M NxN by 



It is straightforward to verify that these equations define a pair of compatible left and right 
actions. The resulting bilateral semidirect product if called the block product of M and N 
and is denoted M □ N. 

Recall that, for a word language L, we denote its syntactic monoid by M(L). 
The following lemma is an adaption of |Str94j . Lemma VI. 1.2. 

Lemma 4.3 Let I be an attribute set, T = {0, 1} 1 , \i G /, L C r + ' + , m > 1. Then 
U\ □ M(L[m]) recognizes exfo M (L)[m]. More precisely, set J = I\{yu} and S = {0,1} J . 
Consider the syntactic homomorphisms 



of L[m] and exfo |t (L)[m] ; respectively. 

Let 7T : Ui □ M(L[m\) — > M(L[m}), (F,n) i— )■ n be the projection homomorphism, and let 
o : S — > T be the alphabet mapping that maps every a G £ to the letter a with a \ J = a 
and a(/i) = 0. We extend a to a homomorphism £ m '* — > Y^ - '* as usual. 
There exist homomorphisms ( : S m, *cr — > U\ □ M(L[m}) and r : Ui □ M(L[m}) — > 
M(exfo M (L)[m]) such that (n = r] L[m] \ (S m a)* and o~(t = r] exf0ii{L)[rn] . 

Proof We define the mapping ®:Ex{0,l}->r the following way: 



(Then a = a <S> for every a G X.) If p G and q G {0, are pictures of the same 

size, we write p £g> q for the equally sized picture over T with {p ® q) (i, j) = p(i, j) eg) q(i, j) 
for every G dom(p). 



(nF)(n 1 ,n 2 ) 
(Fn)(nx,n 2 ) 



F(nin,n 2 ), 
F(n u nn 2 ). 



VL[m] 
^exfo M (L)[m] 



: T m '* ^ M(L[m]), 

: S m '* ^ Af(exfo M (L)[m]) 
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'/exfo M (i)[m] 



□ M(L[m]) 



Em,* _ 
(- pm,* 



7T 



M(L[m]) 



Figure 2: Commutative Diagram for Lemma [4.31 



Choose T C M(L[m]) such that L[m] = Trj L ^ m y Define 

C : £"V -> C/i □ M(L[m]), [ ^ F, J_ j VL[m] , , 

\fl m ) \ \a ri 

where F is defined as follows: for all ni,n2 € M(L[m}), the component F(ni,ri2) € {0, 1} 
is iff there exist b\, . . . , b m e {0, 1} such that b\ . . .b m e 0*10* and 

/ ai ® &i 
*H • : ) r?L[ OT ] -n 2 eT. 
\a m <8> &„ 



We extend £ to a monoid homomorphism £ : (£"V)* — > [7i □ M(L[m]) as usual. 

Then indeed = r)L[ m ] \ (S m cr)*, as claimed in the lemma. 

Let 9 = cr£. Then 9 is a homomorphism £ m >* — > U\ □ M(L[m]). Choose 

K:={GE {0, i}A/(£M)xM(LM) | = o} x M(L[m]) C^D M(L[m]) 

We show that exfo A1 (L)[m] = K9~ x . 

Let p G be a picture of size (m, n), say p = 

appropriate F 1 ,...,F n e {0, l}«(Wxtf(iM) suc h t h at 



0-1,1 ' ' ' a l,n 
Q"m.l ' ' ' Q"m,n 



Then there are 



aid 



ai,i \ II aid 

C=\FiA ; I 



\0"m,l / 



\P"m,lj 



ja m j. 



for every I < n. We have 



| Fx, | Y ) w | ••• | F n , 

\0"m,lj 



&l,n 



VL[r, 
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By induction over n one shows that there exists G G {0, \}M{L[ m \)xM(L[m\) such that 



0-1,1 " " " a l,n 

P 9 = ( G, ( _j_ _j_ J 



with 



G(l,l> = n^i 



0,1,1 ■ ■ ■ \ / Ol^+l • • • fll.n 

ill i ; mm], i ; i 



J=l \ \ <^m,l' • • d m ,i—\ J y Q, m ;_|_i - • • Q, m ,n 

Thus we have the following equivalence chain: 
p6> G if iff G(l, 1) = iff there is I < n such that 



a i,i ' ' ' a i,i-i \ I Oij + i ■ ■ ■ a^ n 

= ^( ( _J_ _J I %[m], I _J ) VL[r, 

Om,l' ' ' Om,,l— 1 / \ ^m,i+r ' ' O m ,n 

iff there is / < n and bi, . . . ,b m G {0,1} such that &i . . . b m G 0*10* and 
alT- • • aTJ^i \ ( a, X)l ®b x \ / aTJ+T- • • \ 

\ \ I VL[m) • i I VL[m] ' \ \ I VL[m] £ T 

Om,l' ' ' 0,m,l—l / \Om, 1 ® b m ) \ Om,l+l' ' ' O m , n J 

iff there exists a picture q of size (m,n) over {0,1} such there is exactly one position 
(k, I) G dom(g) with q(k, I) = 1 and 

a l,l ' ' ' a l,n 

\ ; ® ? ) VL[ m ] g r 

Om,l ' ' ' O m , n 

iff there exists q G Unique^} ^} of size (to, n) such that 

p <8> g G Try-^ = L 

iff p G exfo At (L)[m]. 

Since p was chosen arbitrary, this implies if^ -1 = exfo M (L)[m]. Thus U\ □ M(L[m]) and 9 
recognize exfo M (L)[m\. 

Since the syntactic homomorphism factors through any other homomorphism that recog- 
nizes the same language, there exists a homomorphism r such that 77exfb M (z)[m] = 9t = uQt. 
This completes the proof. □ 

The above lemma and its proof closely follow [Str94], Lemma VI. 1.2. Straubing's lemma 
is obtained from ours by choosing m = 1. Besides, [Str94] does not introduce the operator 
exfo^ from Definition I4.1[ so it must be applied to a formula (f rather than to its language 
L = Mod(f). In |Str94j . that formula is from the first-order theory with order, but that 
condition is of no concern, as the syntax of that formula is irrelevant for the proof. 
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4.5 Using the Group Complexity 

In this section, we finish the proof of Theorem 12.101 A semigroup that does not contain 
a non-trivial group is called aperiodic. A semigroup homomorphism <p> : S — > T is called 
aperiodic if for every aperiodic subsemigroup W of T, its pre-image \Vip~ 1 is aperiodic. 
Recall the definition of group complexity from Section 12 .71 The following theorem is from 
[Rho74bj . see e.g. [RS09] . Theorem 4.9.1. 

Theorem 4.4 (Fundamental Lemma of Complexity) Let S, T be semigroups. Let 
(p : S T be a surjective aperiodic homomorphism. Then c(S) = c(T). 

The observation of the next lemma has probably been made before, but I did not find 
an explicit statement of it in the literature, so we prove it here. A similar statement 
(concerning the Malcev product) can be found in |RS09] . Corollary 4.9.4. 

Lemma 4.5 Let S,T be semigroups such that S is aperiodic. Then c(S **T) = c(T). 
Proof We write S additively and write ns = s + • • ■ + s to denote, for n > and s G S, 

n times 

the n-fold sum of s. By Theorem 14.41 it suffices to show that the projection homomorphism 
7r : S ** T — > T, (s, t) h->- t is aperiodic. 

Let W be an aperiodic subsemigroup of T. Let G be a cyclic group contained in Wn^ 1 . 
It suffices to show that G is trivial. 

Choose g G G such that G is generated by g, i.e., G = {g, g 2 , . . . , #' G '}. Choose s G S and 
t G T such that g = (s, t). We have {t, t 2 , . . . , t |G| } = Gn is a group contained in W, thus 
it is trivial, which implies t = t 2 . Simple induction shows that for every n > it holds 

(a, t) n+2 = ^st + ra(tst) + ts, . 

Since S is aperiodic, there exists n > 1 such that n(tst) — (n+ l)(tst). Now (s,t) n+2 = 
(st + n(tst) + ts, t) = [st + (n + l)(tst) + ts, t) = (s, t) n+3 . Since (s, t) generates the group 
G, this implies that (s, t) is the identity of G, so G is trivial, which completes the proof. □ 

Remark 4.6 Let L be a word language. Let M be a monoid that recognizes L. Then 
c(M(L)) < c(M). 

Proof Set n = c(M). Since M recognizes L, the syntactic monoid of L is a homomorphic 
image of M and hence M(L) -< M. Since M G C n and C n is a pseudovariety, this implies 
M(L) G C n , thus c(M(L)) < n = c(M). □ 

Lemma 4.7 Let / &e an attribute set, let \i G / &e an attribute, let L be a set of pictures 
over alphabet {0, 1} 1 , let m > 1. Tnen c(M(exfo jU (L)[m])) < c(M(L[m])). 
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Proof By Lemma H~3| the monoid UiOM(L[m]) recognizes exfo M (L)[m]. By Remark 14.61 

c(M(exfo M (L)[m])) < c(C/i □ M(L[m])) 

= c([/ 1 M{LH)xM(LH) **M(L[m])). 

Since is aperiodic, so is [/* f ( L H)>< M ( L M)^ Thus the claim follows by Lemma [4.51 □ 

Now we show that the group complexity is not increased by existential first-order quantifi- 
cation, disjunction, and negation. More precisely: 

Proposition 4.8 Let m > 1. For every formula ip, we set M(<p) = M(Modi^K( L p)[ m \) for 
abbreviation, where the attribute sets I, K are understood as containing the indices of all 
free variables (or all free first- order variables, respectively) of (p. 
Let (p, if) be formulas. Then 

c(M(3x vV )) < c(M(tp)), (24) 
c(M((pVip)) < max{c(M(v?)),c(M(^))}, (25) 
c(MHp)) = c(il%>)). (26) 



Proof Ad ( |24|) : This is an immediate consequence of Lemma [4.71 and Remark 14.21 
Ad ()Z5j> : Let n = max{c(M(^)), c(M(^))}. Then M(ip),M(ip) G C n . The direct 
product M((p) x M(ip) recognizes the word language Mod(tp V il>)[m], thus M(y V ■?/>) is a 
homomorphic image of that direct product. Since C n is a pseudovariety, M(<£> V if)) G C n , 
i.e., c(M(y? V ip)) <n. This completes the proof of f[25l) . 

Ad ( I26p : Let n = c(M((p)) Since Modi ^(^f) = Unique IK \Modi i K('p), the word lan- 
guage Mod7- j x( _, V 9 )[ m ] is recognized by the direct product M (Unique T K [m]) x M(<p), thus 
M(-i(p) is a homomorphic image of that direct product. Since M (Unique T K [m)), M(<p) G 
C„ and C n is a pseudovariety, we conclude M(-«p) G C n , i.e., c(M(-xp)) < n. Equality 
follows by symmetry. □ 

Define s : N — > N, s(m) = 2 m . As usual, s°(m) = m and s fe+1 (m) = s(s k (m)) for every 
fc. The function s k is fc-fold exponential. Fix some k > 1 for the rest of this section. 
Furthermore, assume that T = {0, l} 7 is an alphabet with / = J U K for disjoint attribute 
sets J, 

Theorem 4.9 (|M T97j ) Lei ip be a Y,™ on -formula with free set variables in (X M ) Mg j and 
free first-order variables in (x u ) u& k- Then there exists c > 1 swc/i i/iai /or every m > 1 
i/iere exists an NFA with at most s k (cm) states that recognizes Mod^x^)^}- 

The original theorem only states the above result for the case K — 0. The present form 
follows easily by using Modify) = Modi\jKfi(<p) H Unique j ^. 

Let n > 1. The transition monoid (see Section H~2l) of an NFA with n states is a sub monoid 
of the monoid B n of binary relations of an n-set (see Definition 13.141) . 

Proposition 4.10 Let ip be a £™ on -formula. There exists c > 1 stzc/i that for every m > 1 
t/ie syntactic monoid of Mod((p)[m] divides B s kr cm \. 
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Proof Choose c according to Theorem 14.91 Let m > 1. Set n = s k (cm). Let 21 be an 
NFA with at most n states that recognizes Mod(f)[m}. Let M be the transition monoid of 
21. Since M recognizes Mod(if)[m}, the syntactic monoid of Mod((p)[m] is a homomorphic 
image of M, thus M(Mod((p)[m\) -< M. Besides, M -< S„. Since -< is transitive, we have 
M(Mod{(p)[m\) -< 5 n , i.e., the claim. □ 

Proposition 4.11 FO(X™ on ) zs at mosi k-fold exponential wrt. group complexity. 

Proof Let J 7 be the class of formulas <p such that the function m i— > c(Mod((p)[m]) is 
at most fc-fold exponential. By Proposition I4.8[ J 7 is closed under existential first-order 
quantification, disjunction, and negation, i.e., FO(J-") = T . 

By Proposition 14.101 and Theorem I3.18[ the class £™ on is at most A;-fold exponential wrt. 
group complexity, i.e., £™ on C T . Thus FO(£™ on ) C FO(J r ) = J 7 , which finishes the proof. 

□ 

We are now ready to prove the main result of this paper. 

Proof of Theorem I^TOl By Remark EU we have EJ ,on (ng_ 1 (Ai' ,on )) ^ ^T° n (^k°i) ^ 
E mon q FO(S^ on ). Therefore it suffices to show that E5 non (n°_ 1 (A5 non )) is at least fc-fold 
exponential whereas FO(S^ on ) is at most A;-fold exponential wrt. group complexity. The 
first fact is Theorem I3.19[ the second is the preceding proposition. □ 



5 Conclusion 

We continued the work of |Mat99t IMat02] to investigate the expressive power of first-order 
quantifications in the context of picture languages. We have adapted a lemma by Straubing 
that analyses the effect of first-order quantifications in terms of monoid complexity. We 
combined this with the height fragment technique invented in |Gia94I [GRST96] and used in 
the above papers. This allowed to deduce a new separation result (Theorem 12. 101) . It may 
be stated informally as: Adding one more set quantifier alternation gives you expressive 
power that cannot be captured by adding any number of first-order quantifier alternations. 

At the same time we have found a new sequence of picture languages that witness the 
strictness of the quantifier alternation hierarchy of monadic second-order logic. Unlike the 
picture languages in |MT97j . these new witness picture languages are not characterized by 
the sizes of their pictures, but rather by the group complexity required to recognize them. 

5.1 Remarks on the Height Fragment Technique 

The height fragment technique (Remark 12. 91) plays a crucial role for the separation results 
for picture language classes defined by quantifier alternation classes of monadic second- 
order logic. Therefore, it may be instructive to summarize the measures that have been 
considered so far (more or less explicitly) in the literature and in this paper. 
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• The state set size measure assigns to every regular word language L the minimal 
number of states of an NFA that accepts L. 

• The singleton length measure assigns to every singleton word language {a n } over a 
singleton alphabet {a} the length n of its only element. 

• The minimal length measure assigns to every non-empty word language over a sin- 
gleton alphabet the length of its shortest element. 

• The group complexity measure assigns to every non-empty word language L the group 
complexity of its syntactic monoid. 

The proof that the class of E™ on -definable picture languages over {0, 1} is not closed under 
complement was done in [GRST96J and uses the state set size measure. That picture 
language class is at most singly exponential wrt. state set size, but the class of Il l 1 nc ' n - 
defmable picture languages is not, as it contains a language with state set size 2 Q,( - m \ 
namely the picture language of all pictures of the form pp, where p is a non-empty square 
picture. 

The first result involving the singleton length measure is from |Gia94j and says that the 
class of recognizable (or, by [GRS T96] equivalently, of E^-definable) picture languages 
over a singleton alphabet is both at most and at least 1-fold exponential. 
Generalizing Giammarresi's result, in |MT97] (and |Sch97j . respectively) it is shown that 
the class of £™ on -definable picture languages over a singleton alphabet is at most (and at 
least, respectively) A;- fold exponential wrt. singleton length. 

In [Mat99j . Corollary 3.66 and Theorem 4.25, it is shown that FO(X™ on ) is both at most 
and at least (fc+l)-fold exponential wrt. singleton length. 

In |Mat99j . Theorem 3.61 and Corollary 4.15, it is shown that IT™ on is at least (AH-l)-fold 
exponential whereas £™ on is at most /c-fold exponential wrt. minimal length. This allowed 
to separate these two classes for the case of a singleton alphabet. 

Our result Corollary 12.111 is based on the group complexity measure. That corollary can 
be proved neither by state set size nor by singleton length, since for every k > 1, even 
the class of FO(T,™ on )-definable picture languages is at least /c-fold exponential wrt. state 
set size ( |MT97j ). and both FO(S^ on ) and £™°" are both at most and at least (fc+l)-fold 
exponential wrt. singleton length ([Mat99j). 

5.2 Open Questions 

Corollary 12.111 states that there is a £™°"-definable picture language that is not FO(£™ on )- 
definable. Lemma 13.131 shows that the alphabet is {0,l}^- mp ' end \ i.e., the alphabet size 
is four. We note without proof that one can reduce the size of the alphabet to two by 
applying standard encoding techniques. It remains open whether we can even reduce the 
size to one. For that case, we only know: 

Theorem 5.1 ( |Mat99| . Theorems 2.29, 2.30) Let k > 1. For a singleton alphabet, 
there is a T>™°"- definable picture language that is not FO(£™°") -definable. 
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The following problem from |Mat99l IJMOlj remains open, too. 

Problem 5.2 Is there is an MSO-formula that is not equivalent to any S! ] non (FO(S l 1 non ))- 

formula? 

Separation results such as Corollary 12 . 1 11 may be transferred to other classes of structures, 
for example, to directed graphs, using standard encoding techniques. This has been carried 
out formally in [Mat99j, Chapter 5. 

Problem 15.21 is also open in that more general setting of directed graphs. Answering 
that question might be the first step towards attacking the closed hierarchy as defined in 
[AFSOOj . 

I find the following questions interesting from a methodological point of view: 

Problem 5.3 Is there any separation result concerning quantifier alternation classes of 
MSO-formulas ( as defined in Section \2.5\) that either holds for directed graphs but not for 
pictures, or holds for pictures but cannot be proved with the height fragment technique? 

6 Acknowledgments 

Thanks go to several participants of the 2010 workshop on "Circuits, logic, and games" in 
Dagstuhl: Wolfgang Thomas for discussions on the draft of this paper; Nicole Schweikardt 
for her detailed analysis and helpful feedback; Jean-Eric Pin for the copy of |Pin86j and 
for pointing me helpful hints when I was stuck in the proof of Proposition 14.111 — later 
I discovered that the group complexity was more suitable than the investigation of the 
symmetric subgroups, so the respective papers are not cited; Klaus- Jorn Lange for a proof 
related to the block product, even though that proof did not make it into the present 
paper because I discovered later that I had asked the wrong question; Pierre McKenzie for 
pointing me to the Landau function, which I did not need in the end either, because the 
lengths of the cyclic subgroups then seemed to be the wrong trace; and Thomas Colcombet 
and Etienne Grandjean for discussions on related subjects. 

Last, but not least, I am grateful to Thomas Wilke. When he pointed me to Chapter VI 
in [Str 94j . it was not the first time that he gave me the crucial hint that I needed to turn 
a vague idea into a theorem. 

References 

[AFS00] M. Ajtai, R. Fagin, and L. Stockmeyer. The closure of monadic NP. Journal of Computer and 
System Sciences, 60(3):660-716, 2000. Journal version of STOC'98 paper. 

[Eil76] S. Eilenberg. Automata, Languages, and Machines. Number Bd. 2 in Pure and applied math- 
ematics. Academic Press, 1976. 

[Gia94] D. Giammarresi. Two-dimensional languages and recognizable functions. In G. Rozenberg and 
A. Salomaa, editors, Developments in Language Theory, Proceedings of the conference, Turku 
(Finnland) '93, pages 290-301. world scientific, Singapore, 1994. 



REFERENCES 



37 



[GRST96] D. Giammarresi, A. Restivo, S. Seibert, and W. Thomas. Monadic second-order logic and 
recognizability by tiling systems. Information and Computation, 125(1) :32-45, 1996. Journal 
version of STACS'94 paper. 

[JM01] D. Janin and J. Marcinkowski. A toolkit for first order extensions of monadic games. In Pro- 
ceedings of 18th Annual Symposium on Theoretical Aspects of Computer Science (STACS'01), 
volume 2010 of Lecture Notes in Computer Science, pages 353-364. Springer- Verlag, 2001. 

[KR65] K. Krohn and J. Rhodes. Algebraic theory of machines. I. Prime decomposition theorem for 
finite semigroups and machines. Trans. Amer. Math. Soc, 116:450-464, 1965. 

[Mat99] O. Matz. Dot-depth and monadic quantifier alternation over pictures. PhD thesis, RWTH 
Aachen, 1999. 

[Mat02] O. Matz. Dot-depth, monadic quantifier alternation, and first-order closure over grids and 
pictures. Theor. Comput. Sci., 270(l-2):l-70, 2002. 

[MST02] O. Matz, N. Schweikardt, and W. Thomas. The monadic quantifier alternation hierarchy over 
grids and graphs. Information and Computation, 179(2):356-383, 2002. 

[MT97] O. Matz and W. Thomas. The monadic quantifier alternation hierarchy over graphs is infinite. 

In Twelfth Annual IEEE Symposium on Logic in Computer Science, pages 236-244, Warsaw, 
Poland, 1997. IEEE. 

[Pin86] J.-E. Pin. Varieties Of Formal Languages. Plenum Publishing Co., 1986. 

[Pin96] J.-E. Pin. Logic, semigroups and automata on words. Annals of Mathematics and Artificial 
Intelligence, 16:343-384, 1996. 

[Rho74a] J. Rhodes. Finite binary relations have no more complexity than finite functions. Semigroup 
Forum, 7:92-103, 1974. 

[Rho74b] J. Rhodes. Proof of the fundamental lemma of complexity (strong version) for arbitrary finite 
semigroups. J. Comb. Theory, Ser. A, 16(2):209-214, 1974. 

[RS09] J. Rhodes and B. Steinberg. The q-theory of finite semigroups. Springer monographs in math- 
ematics. Springer, 2009. 

[RT89] J. Rhodes and B. Tilson. The kernel of monoid morphisms. Information and Computation, 
62:227-268, 1989. 

[Sch97] N. Schweikardt. The monadic quantifier alternation hierarchy over grids and pictures. In Mo- 
gens Nielson and Wolfgang Thomas, editors, Computer Science Logic, volume 1414 of Lecture 
Notes in Computer Science, pages 441-460. Springer, 1997. 

[Str94] H. Straubing. Finite automata, formal logic, and circuit complexity. Birkhauser Verlag, Basel, 
Switzerland, 1994. 



